Damaged Merchandise? A Review of Experiments That Compare Usability Evaluation Methods

An in ter est in the de sign of in ter faces has been a core topic for re search ers and prac ti tio ners in the field of hu man–com puter in ter ac tion (HCI); an in ter est in the de sign of ex per i ments has not. To the ex tent that re li able and valid guid ance for the for mer de pends on the re sults of the lat ter, it is nec es sary that re search ers and prac ti tio ners un der stand how small fea tures of an ex per i men tal de sign can cast large shad ows over the re sults and con clu sions that can be drawn. In this re view we ex am ine the de sign of 5 ex per i ments that com pared us abil ity eval u a tion meth ods (UEMs). Each has had an im por tant in flu ence on HCI thought and prac tice. Un for tu nately, our ex am i na tion shows that small prob lems in the way these ex per i ments were de signed and con ducted call into se ri ous ques tion what we thought we knew re gard ing the ef fi cacy of var i ous UEMs. If the in flu ence of these ex per i ments were triv ial, then such small HUMAN-COMPUTER INTERACTION, 1998, Volume 13, pp. 203–261 Copy right © 1998, Law rence Erlbaum As so ci ates, Inc.